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FOREWORD 



This booklet has been prepared as part of a project sponsored by the 
United States Education Department (USED) on evaluation in early childho9d 
Title I (ECT-I) programs. It is one of a series of resource books developed 
in response to concerns expressed by state and local personnel about early 
childhood Title I programs. The series describes an array of diverse 
evaluation activities and outlines how each of these might contribute to 
improving local programs. The series revolves around a set of questions: 

• Who will use the evaluation results? 

• What kinds of information are users likely to find most helpful? 

• In what ways might this information aid in program improvement? 

• Are the potential benefits substantial enough to justify the cost 
and effort of evaluation? 

Together, the resource bocks address a range of issues relevant to the 

evaluation of early childhood programs for educationally disadvantaged 

children. The series comprises the following volumes: 

• Evaluating Title I Early Childhood Programs: An Overview 

• Assessment in Early Childhood Education 

• Short-Term Impact Evaluation of Early Childhood Title I Programs 

• An Introduction to the Value-Added Model and Its Use in Short-Term 
Impact Assessment 

• Evaluation Approaches: A Focus on Improving Early Childhood Title 
I Programs 

- • Longitudinal Evaluation Systems for Early Childhood Title I Programs 

• Evaluating Title I Parent Education Programs 

The development of this series follows extensive field work on ECT-I 
programs (Yurchak 5 Bryk, 1979). In the course of that research, we 
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identified a number of concerns that SEA and LEA officials had about ECT-I 
programs ,_and the kinds of information that might be helpful in addressing 
them. Each resource book in the series thus deals with a specific concern 
or set of ccncerns. The books and the evaluation approaches they describe 
do not, however, constitute a comprehensive evaluation system to be uniformly 
applied by all. Our feasibility analysis CBryk, Apling, § Mathews, 1978) 
indicated that such a system could not efficiently respond to the specific 
issues of interest in any single district at any given time. Rather, LEA 
personnel might wish to draw upon one or more of the approaches we describe, 
tailoring their effort to fit the particular problem confronting them. 

Finally, the resource bcoks are not comprehensive technical manuals. 
Their purpose is to help local school personnel identify issues that might 
merit further examination and to guide the choice of suitable evaluation 
strategies to address those issues. Additional information and assistance 
in using the various evaluation strategies are available in the more techni- 
cal publications cited at the end of each volume, and from the Technical 
Assistance Centers in the ten national regions. 
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I. BACKGROUND AND RATIONALE 

Program evaluation has been a requirement under Title I since its in- 
ception. Over the years, most evaluations have concentrated on the account- 
ability question: Has the program achieved its goals? This information, 
required for local educational agencies' (LEAs) annual funding applications, 
is potentially useful to school administrators, school boards, and Parent 
Advisory Councils (PACs) in examining tOie overall effectiveness of school 
programs. The Department of Education CSJ>) proposed evaluation models for 
grades 2 through 12, and comparable procedures for early childhood programs 
have been designed to generate such information Csee Tallmadge § Wood, 1978, 
and the res -urce book Short -Tera Impact Evaluations of Early Childhood Title 
I Programs by Haney, 1980) . 

Recently, attention has begun to focus on how the results of evaluation 
might lead to improved program practice. This is reflected in the Education 
Amendments of 1978, which stipulate that Title I evaluations should provide 
LEA staff with information for planning and improving their projects and 
activities. In the Huron Institute's visits to early childhood Title I 
CECT-I) programs across the country, we found a number of school districts 
that had developed their own evaluation activities to address local con- 
cerns about improving ECT-I programs, and other districts that were inter- 
ested but did not know quite how to proceed. 

Unfortunately, the evaluation models for grades 2 through 12 and the 
early childhood equivalents were designed primarily to generate data about 
overall program effectiveness. As a result, they often will not provide 
sufficient detail to LEA staff concerned about where a program may be 
faltering, why problems are occurring, and how improvements can be made. 
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This resource book is a first response to that need. We survey here a wide 
range of evaluation techniques that can generate quite extensive information 
about programs, which can be helpful in suggesting ways to improve. In 
choosing techniques to discuss, we have drawn both on methods that were 
used in ECT-I programs we visited and on other promising approaches. 
Some of these methods can also produce information useful for school district 
policy formulation and accountability, or data about individual children that 
teachers can use to make instructional decisions; we present them here be- 
cause they seem to us especially useful to local staff as they attempt to 
improve their programs. While all the methods are intended to inform efforts 
to improve programs, each addresses somewhat different questions; thus their 
usefulness will vary from context to context. 

In describing these evaluation methods, we try to suggest the kinds of 
question that each can address, the kinds of information that it might pro- 
duce, and the resources and personnel commitments that it requires. Each of 
the methods we discuss, however, is not so much a precise and well defined 
set of activities as a general approach to inquiry, All have considerable 
merit and no one approach is unequivocally best. For an evaluation to 
generate useful information for program planning and improvement, it must 
be tailored to local concerns and the nature of the particular program. 
Although we introduce each as a separate method, actual evaluations might 
consist of a variant on one approach or some combination of approaches. 
These may be used to look at an ECT-I program in its entirety, or perhaps at 
just one program component or particular feature. In short, the possibil- 
ities are enormous. Our modest goal for this resource book, then, is to 
suggest some of the possibilities, but not to fully map the terrain. 

ERIC 



LEA program staff will usually find it necessary to draw on evaluation 
expertise in designing and implementing such a study. While some LEAs " 
have an evaluation staff to support these efforts, many may want to seek 
outside help, for example from the Title I Technical Assistance Centers (TACs) 
or from local evaluation consultants. This document is not a substitute for 
professional assistance. Rather, we intend it as an idea book for local 
program staff on how evaluation can help them to plan and improve programs. 

We have organized the evaluation methods described in this resource 
book into three basic types: 

• Program review methods such as self-study and outside review 

• Quantitative social science research methods that use student test 
data and program descriptive information 

• Methods that are generally referred to as naturalistic or qualita- 
tive inquiry. 

The following three chapters describe these three types of methods and illus- 
trate them with short examples. For each type, we discuss human resource 
requirements, the procedures to be used, and some situations where the methods 
might be most useful. In the concluding chapter, we summarize the alter- 
natives presented and raise a number of issues that LEAs should consider 
about the process of developing and implementing an evaluation plan. 
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II . PROGRAM REVIEW METHODS 



Both self-study and outside review axe ways of examining ECT-I programs 
that feature the full involvement in the evaluation process of teachers and 
other program staff from the school system. ECT-I staff can examine both 
general program features such as program goals, parent participation, and 
staff qualifications, and specific activities such as teaching practices and 
selection procedures. Although the two methods are similar, the audiences 
for the evaluations may differ depending on whether the program staff or an 
outside team evaluates the program. 

In a self-study, staff members assess their own program. Working in 
committees, they identify strengths and weaknesses and make recommendations 
for improvement. Not only do ECT-I staff members conduct the evaluation, 
but they often become the chief audience for the results. Although the 
school board, local administrators, and parent groups might receive the 
final report on the self-study, it is the staff that is most likely to use 
the results of the evaluation to improve program practice. 

In an outside review, a visiting committee, usually consisting of staff 
from nearby ECT-I programs and early childhood education specialists, examines 
a program's goals, resources, materials, procedures, personnel qualifications, 
and facilities. As in self-study, the committee looks for strengths and 
weaknesses in each area and recommends improvements. Unlike self-study, 
which aims primarily at self-examination and self- improvement, outside re- 
view may also treat accountability considerations, for example in program 
accreditation. Thus, while ECT-I staff members are often the main users 
of the results of self-study, outside review may be aimed more at parents, 
program administrators, and school boards. 
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AN EXAMPLE: OUTSIDE REVIEW OF A SPECIAL EDUCATION PROGRAM a 

A state director of special education and his staff initiated an outside 
review to identify exemplary projects and to provide local program staffs 
with suggestions for improving their practice • Each visiting team was com- 
posed of representatives of the state division of special education, special 
education administrators and teachers, regular education staff, and parents . 
In looking at the special education program in one community, the team ex- 
amined the following areas: 

• The preschool component 

• Kindergarten screening 

• Staff development 

• Program delivery * 

• Public communications 

• Transportation 

• Physical facilities. 

The team reviewed data in student records, interviewed staff, and had parents 
fill out questionnaires. For each area, the team identified commendable 
practices, pointed out problems, and recommended changes. For example, the 
team reported no problems with the preschool component, praising it for the 
introduction of preschool labs, the integration of normal and handicapped 
preschool children, and t" e exemplary efforts to locate preschool children 
with special needs. It also commended kindergarten screening but noted one 
problem: principals and regular kindergarten teachers did not see the value 
of the screening procedures . The team therefore recommended that these 
groups be trained in the use of screening instruments and be involved in 
the actual screening* The full report was distributed to the project dixec- 
tor, staff members, and parents. 

/ ^ 

ERIC / 

/ 



-7- 



HUMAN RESOURCE REQUIREMENTS 
Self-Study 

Staff members are the chief participants in self-study. A steering 
committee is drawn from the staff to organize and coordinate .He evaluation. 
This group is the keystone of the self-study. It appoints other staff mem- 
bers to various subcommittees, assists the subcommittees in their work, calls 
and conducts general meetings to examine and approve reports , and prepares 
the final report. Each subcommittee gathers information on some specific 
program areas, assesses strengths and weaknesses, suggests changes, and re- 
ports its findings to all participants in the study. 

After a subcommittee has assembled its results and made recommendations, 
the entire study staff might meet to discuss the report, ask for clarification, 
suggest revisions, and approve the report. When all subcommittee reports 
have been approved, the steering committee assembles the final report and 
distributes it to the appropriate audience. This may include the program 
director, the staff, the school board, parents, and other interested members 
of the community. 

For smaller ECT-I programs, or if only a few program areas are to be 
examined, a hierarchical structure for self-study may be unnecessary. In- 
stead, the whole staff and the director might make up a single committee to 
identify and study a few key areas. For example, they might decide to assess 
program goals, chief instructional areas, and staff qualifications. At some 
other time, they might evaluate other areas such as program administration 
and facilities. 

The continuing involvement of the program staff in the evaluation is 
the chief strength of self-study. If staff members see a real need for 
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improvement because of their participation in the study, important changes 
can result. Self-study can provide a strong link between evaluation results 
and actual program modification, because those who conduct the evaluation 
also play central roles in implementing proposed changes. 
Outside Review 

Outside review procedures also depend on practitioners. As the example 
cited earlier illustrates, a visiting team may include teachers and adminis- 
trators from nearby programs, early childhood experts, and state ECT-I staff, 
who are assigned to subcommittees correponding to their interests and ex- 
pertise* For example, an ECT-I program director might concentrate on assess* 
ing the administration of the progTam, while a specialist in early childhood 
language development would focus on the program's language arts activities. 

In planning for an outside review, a staff member or committee is usually 
designated to coordinate the visit, arrange interviews with the staff, plan 
tours of facilities, and organiz^ classroom observations. A chairperson is 
generally selected from among the\ visiting committee to oversee activities 
before, during, and after the visitt. This person also reports orally to the 
program staff at the end of the visit and drafts the final report, s ummar izing 
the committee's views and recommendations. It is often useful to precede an 
outside review with a self-study, because the resulting report can help to 
focus the inquiry. 

Like self-study, outside review can be relatively inexpensive, depending 
upon the extent to which committee members are remunerated, if at all. More- 
over, it can provide a fresh and different perspective on the program that 
internal reviews might miss, and lead to informal exchanges of information 
as visitors report on their experiences and those of other programs they 
have visited. 

14 
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PROCEDURES FOR CONDUCTING SELF-STUDY AND OUTSIDE REVIEW 
Self-Study 

Self-study data collection depends on the work of several subcommittees, 
each focusing on a specific part of the program and seekin* answers to cu--- 
tions related to that part. For example, the subcommittee examining program 
structure might raise questions such as: What is the staffing of the program? 
How is the typical day organized? To what extent and how are individual 
programs provided for each child? 

Identifying questions to guide the self-study, and refining this list, 
is an important first step. A self-study can easily go awry by spending too 
much time seeking information to "answer" unimportant questions or examining 
areas in which a program change is simply not possible. In this regard, it 
can be helpful to suggest possible findings from each question before con- 
ducting the study, and thea examine whether these lead in some productive 
direction. If it turns out that, regardless of how a particular question 
might be answered, no practical action could result, examining this question 
in the self-study would be fruitless. 

Subcommittees may use several data gathering methods in conducting this 
work. T. r can rev i ew documents such as program applications, evaluation 
reports, and curriculum guides. They can interview the program director, 
principals, and other staff members. They can distribute questionnaires to 
teachers in the program. While thes; data collection activities are guided 
by the list of identified questions, that list may change somewhat as the 
study progresses. 

As information is gathered, it is discussed at committee meetings. 
The committee members for each self-study area examine the results of their 
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questionnaires, interviews, and documen 1 " reviews. They may look for agree- 
ment and disagreement among different sources of information- They may also 
give their own opinions about program strengths and weaknesses. As this in- 
formal analysis proceeds, they discuss what they think they have found and 
work to reach consensus on answers to specific questions. If consensus does 
not emerge, the alternative points of views then are documented. Finally 
th<* chairperson writes a report assessing the current state of the program 
in the areas examined, recommending improvements and summarizing the findings. 
Outside Review 

In preparing for a site visit, the visiting committee examines avail- 
able documents such as grant proposals, program descriptions, curriculum 
material, budgets, evaluations, and self-study reports. On that basis, com- 
mittee members identify issues and questions to be addressed. During the 
visit, these are investigated through interviews and observations. Inter- 
views are usually unstructured and short because of time constraints on the 
committee and program staff. They might include some brief questioning of 
teachers following a classroom observation, and more extensive discussions 
with the program director, principals, and parents. Because committee mem- 
bers cannot participate in day-to-day program activities, a considerable 
amount of information is drawn from passing observations as they interview 
staff members, attend meetings, or just roam the halls. 

Like self-study, outside review depends on informal data analysis. If 
a self-study or other evaluations have been done, the visiting committee can 
consider them and try to build on the findings. Alternatively, the committee 
might compare the current state of the program, as determined by observations, 
interviews, and program documents, to some set of criteria for a "good" 
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early childhood program. These criteria may be drawn from program guide- 
lines, from a state education agency, or from material generated by commer- 
cial program developers. 
FURTHER CONSIDERATIONS 

Self-study requires a gTeat deal of staff time because of the heavy in- 
volvement of staff in the actual evaluation process. Unless staff members 
understand what will be required of them, and how the evaluation will serve 
to improve the program, they may be reluctant to devote the time and energy 
necessary to make self-study successful. 

The data collection procedures for a self-study need careful attention. 
It is important to decide in advance what questions to ask about each com- 
ponent of the program. A first step might be to consult a document such as 
the Elementary School Evaluation Criteria (National Study of School Evalua- 
tion, 1973) for guidelines or starting points for self-study, and to modify 
these as necessary for each specific evaluation. A booklet aimed at elemen- 
tary schools, for example, may not deal with important early childhood con- 
cerns such as screening or parent involvement. At the same time, it will 
include areas that may be beyond the scope of an ECT-I evaluation, like 
transportation and support services. 

Another self-study issue is the depth and objectivity of data collection 
and analysis. It is crucial that examination of the program be frank and 
self -assessment candid. Depending upon who is actually involved in the 
self-study, the results may be biased in their favor, while shortcomings 
may be blamed on others, such as administrators, parents, pupils, and the 
community. Exercising care in planning for the self-study, and in selecting 
members for each committee, can alleviate these problems. 
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In outside reviews, because the visits are short, data collection and 
analysis may not be as thorough as one would like. Thus important program 
aspects may be missed. Brief sporadic visits to evaluate classrooms have 
several potential drawbacks. The sample of classroom activities observed 
may be small and atypical. Teachers and children know that visitors are 
coming, so that special preparations may have been made. Teachers try to 
exhibit their best practices, and young children may be distracted by a 
group of strangers. Moreover, the observers 1 views of teachers and class- 
rooms differ from children's views; yet the latter may be the most informa- 
tive for discovering ways to improve the progTam. Combining brief class- 
room observations with follow-up teacher interviews can help to identify 
these problems of perspective. As in the self-study, care should be exer- 
cised in the selection of outside reviews in order-tfo obtain a diverse but 
balanced set of perspectives. CFor further discussion of these points, see 
Scriven, 1977.) 

These cautions should not discourage the use of self-study and outside 
review. The results of such an effort can be quite suggestive, and lead to 
new programmatic or further evaluation activities. As we shall see, no pro- 
cedure is without fault, and no method by itself can ensure progTam improve- 
ment. Rather, it seems to us that strong staff interest in conducting the 
study and good local technical assistance in its design and implementation 
are critical to successful evaluation and use of the results. 
NOTE 3 ON SOURCES OF FURTHER INFORMATION 

There are several guides and manuals for conducting self-studies and 
outside reviews. Among these is material produced by the National Study of 
School Evaluation (1973) , the National Council for Accreditation of Teacher 
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Education (1970), the National Association of Secondary School Principals 
(Georgiades, 1978), and the Arkansas Elementary School Council (1971). Of 
course, these sources can only guide .the development of a self-study and 
outside review protocol for ECT-I programs; each was written for different 
purposes and hence must be adapted for early childhood programs 

In addition, other sources mentioned throughout this resource bwok will 
help direct different aspects of an ECT-I program review. Payne (.1951) has 
written a classic book on framing and asking questions. Ferreira and Burges • 
(1976) and Boehm and Weinberg's (1977) books on classroom observation are 
practical and concise. Books by Kerlinger (1973, Chapters 28 and 31), 
Travers (1978, Chapters 8 and 12), and Tuckman (1972, Chapters 7 and 18) 
have sections on interviews and observation, which should be helpful to 
anyone planning and carrying out self-studies and outside reviews. 



V) 



9 

ERIC 



-1S- 



III. QUANTITATIVE METHODS 

We have grouped together in this section three evaluation methods based 
on quantitative social science research techniques: 

• Structured process evaluation, which focuses on how services are 
delivered 

• Product evaluation or examination of program outcome data, which 
considers the impact of services on specific program objectives 

• Process -product evaluation, which combines aspects of the other 
two to analyze the link between program practice and program out- 
comes . 

STRUCTURED PROCESS EVALUATION 

Structured process evaluation examines aspects of how ECT-I services 
are actually delivered and how well this fits with what the program was de- 
signed to do. Such an evaluation might, for example, describe general 
features of a program, such as the number and kinds of students served, 
. what materials are available, and how classrooms are arranged. It can also 
Frovide some fine-grained information about the frequency and duration of 
specific activities and behavior, such as the incidence of children's ag- 
■ gressive acts or the time teachers spend in direct reading instruction. 
When evaluators and ECT-I staff compare these data on classroom practices 
with the program objectives, they may uncover some areas of discrepancy. 
Such information can be helpful in identifying aspects of program practices 
where attempts at change are warranted. 
PRODUCT EVALUATION 

Product evaluation builds naturally on a short-term program impact 
evaluation that might be conducted for purposes of accountability (see the 
resource book Short-Term Impact Evaluations of Early Childhood Title I 
Programs by Haney, 1980). Here, however, the focus is on a more detailed 
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analysis of where and with whom the program succeeds ox does not succeed. 
Such an evaluation might examine, for example, the degree to which certain 
program objectives are achieved or the variability in program effectiveness 
across classrooms, schools, or subpcpuiations of children* If adequate 
data exist on program outcomes, careful examination can indicate the need ' 
for improvement with respect to specific objectives, program sites, or groups 
of children. In addition, product evaluation can provide guidance on where 
to focus efforts for change, 
PROCESS-PRODUCT EVALUATION 

Process -product evaluation combines elements of both of these methods to 
investigate the links between specific program practices (e.g., the amount 
of time children spend in reading instruction) and child outcomes such as 
norm- or criterion-referenced test scores* This method, by examining program 
practices jointly with their consequences, cat* suggest direct ways of im- 
proving the pvogram* Of the three techniques discussed in this chapter, 
process -product evaluation offers perhaps the greatest potential for program 
improvement. It is, however, the most complicated and difficult to imple- 
ment successfully* 
EXAMPLES OF QUANTITATIVE METHODS 
A Structured Process Evaluation* 

A short-term impact evaluation of an early childhood program indicated 
that students were not performing as well as expected* The local director 
of evaluation, drawing on recent research by Berliner and Rosenshine (1976), 
suggested to the program staff that these results might be related to the 

* This case was adapted from several sources: Issac (1977), Fenstermacher 
(1977), and Behnke, Bennett, Chase, Day, Lazar, and Mittleholtz (*o date)* 
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tine students actually spent on learning basic skills. After a series of 
discussions among the evaluation and program staffs, together they decided 
to focus further evaluation on the frequency of classroom disruptions, their 
apparent causes, and how teachers cried to deal with them. 

The evaluation staff assembled eight observers in four teams of two. 
For six weeks these teams observed eight classrooms, using a checklist de- 
veloped. by Berliner and Rosenshine to code the type and number of disruptions 
and teachers' ways of coping with them. The observers also noted how effec- 
tive each coping technique was in terms of how much instructional time was 
lost, whether teaching was interrupted, and how long the disruption continued 
after the technique was used. 

The evaluators found that most disruptions were caused by the students 
themselves, that they usually occurred during reading instruction, and that 
a few children were mainly responsible. Nonverbal techniques, such as iso- 
lating the students who were causing the disruptions, were found to be par- 
ticularly effective in reducing the loss in instruction time. 
A Product Analysis* 

The coordinator of <i pilot preschool program approached the field re- 
search center of a nearby college for advice on how to evaluate their pilot 
effort. A research associate at the c inter persuaded him to develop a 
criterion-referenced test linked to the objectives of the program. Such a 
test, she suggested, could provide three kinds of information: individual 
c^ild data to help in monitoring each child's performance, classroom data 
to assist teachers in focusing efforts on areas needing improvement , and 

* This example is fictional, although the original idea came from Airasian 
and Madaus (1972), and the format of the tables :rom Morris and Fit:- 
Gibbon (1978, pp. 136-143). 
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more general program data to enable the director and staff to identify weak 
program components . After several meetings to clarify program objectives, 
a criterion-referenced test was produced. 

The results from this testing produced considerable information. Table 
1 shows some of the data that could be provided to the classroom teacher. 
From this, the teacher can see, for example, that the first child has done 
well on test items measuring all four objectives and should move on to other 
work, while Larry, the second child, appears to need more instruction in 
all areas. Table 2 shows results from the testing summarized by classroom. 
In this display, teachers can see overall strengths and weaknesses in their 
classes, and so can behave accordingly. For example, the teacher in Class- 
room 1 can easily see that more effort should focus on Objective 4, capital 
letter recognition; and that, while certain children need help in learning 
to write their names, no overall class time need be devoted to this objective. 
Finally, Table 3 provides summary information on the whole program. From 
this, the program director can readily see that Objectives 3 and 4 are not 
being m<st, and this suggests a further examination of instructional activities 
in this area. 

A Process -Product Evaluation 

Some of the cost of day care in this country is subsidized by the 
federal government. In order to assure that adequate care is provided, 
centers that receive such subsidies must meet certain standards of physical 
plant and human resources. Of particular concern were the regulations govern- 
ing staff-child ratio, staff training, and staff experience. To help 
determine the appropriateness of these standards, the Administration for 
Children, Youth, and Families (ACYF) , the federal agency that helps to 
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Table 1 

Results from One ECT-I Classroom Showing 
Which Objectives Each Child Achieved 
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Objective 2 Objective 3 


Objective 4 
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Objective 4 


ECT-I- Classroom 
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99% 
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70% 35% 
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65% 35% 


20% 






Table 3 






Overall Percentage of Students 






Achieving Each Objective 






Objective 1 


Objective 2 Objective 3 


Objective 4 




90% 


70% 41% 


22% 


24 



-20- 



develop the regulations, decided to explore two issues: (1) how these 
standards influence the behavior of workers in the day care centers, e.g., 
how much time is spent teaching children; and (2) how this in turn affects 
children's behavior and development. ACY? contracted with Abt Associates 
to investigate these questions. 

Developing procedures for collecting data on each of these elements 
was the first major task of the Abt staff. To obtain background data on 
day care workers, the staff in a sample of centers were asked to complete 
a questionnaire. Other data, such as staff-child ratio, were collected 
during several observations made in the same centers. Information on the 
behavior of caregivers and children was gathered using structured observation 
protocols. Finally, to assess the children's academic progress, the research 
team administered the Caldwell Preschool Inventory and the Peabody Picture 
Vocabulary tests, first in the fall and then again the following spring. 

These procedures generated a large amount of data, which were analyzed 
with sophisticated statistical techniques. The analysis identified group 
size—the number of children present— as an important feature of the day 
care center. Group size was strongly related to how caregivers spent their 
time: for example, in smaller centers caregivers spent considerably more 
time interacting with children than in larger centers. Amount of inter- 
action, in turn, was strongly associated with positive child experiences 
(e.g., more time engaged in mastery tasks) and better academic development 
(i.e. larger gains on the standardized tests). These results suggested to 
ACYF that future day care center regulations should give more attention to 
group size. 
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Coupon Features 

Each of the examples discussed above represents a highly focused fora 
of inquiry. Each research process began as a result of some pertfeived prob- 
lem. In the first example of structured process evaluation, for instance, 
concern arose about the program's effectiveness. Out of this, some fairly 
specific research questions emerged. Discussion between program staff and 
evaluators, in that example, led to the decision to focus on classroom 
disruptions, their apparent causes, and teachers' attempts to deal with 
them. An appropriate evaluation plan could then be developed^ 

A key feature in developing tha evaluation plan, as seen in each of 
the cases desciibed above, is the selection, or if necessary the creation, 
of suitable data collection procedures for each area of study. In the 
product analysis example, this involved the creation of a criterion-refer- 
enced test (CRT) that was geared to the objectives of the new curriculum. 
In general, the quantitative methods described in this section all rely on 
standardized instruments -especially structured interviews, structured 
observations, and standardized tests. 

Conducting these evaluations often requires time from program personnel 
and extensive support from evaluation staff. This collaboration is needed 
at each step of such a study-beginning with the identification of concerns 
and development of the evaluation plan, through the collection and organiza- 
tion of data, and concluding with data analyses and report writing. In 
principle, quantitative methods can be very efficient evaluative tools when 
the program manager or policy maker wishes to undertake focused inquiry 
into issues that have been fairly well specified in advance. 
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HUMAN RESOURCE REQUIREMENTS 
Structured Process Evaluation 

Structured process evaluation usually requires the participation of 
a program evaluator or evaluation team. In our example, several persons 
were involved. They created or selected appropriate evaluation instruments, 
trained interviewers and observers, directed the evaluation, and wrote the 
reports. Often, the evaluator may participate directly in data collection 
by observing classrooms, interviewing teachers, aides, and parents, and 
distributing questionnaires. In small LEAs with limited resources, a part- 
time person can carry out the entire evaluation, from data collection to 
report writing. 

In general, structured process evaluation depends on a knowledge of 
survey vesearch methods for developing an efficient plan for data collection, 
devising valid interview guides and observation protocols, and carrying out 
the data collection. The quality of this kind of study in large measure 
depends upon the adequacy of the data collection instruments. In some LEAs, 
local evaluation staff in collaboration with program staff have the capacity 
to develop such instruments. Elsewhere, help can often be obtained from 
outside sources. 

Some care also needs to be exercised in selecting the staff who will 

I 

collect the data. Interviewers should be able to put respondents at ease 
so as to encourage free and full discussion. Moreover, they must be able 
to record responses without showing surprise, approval, or disagreement. 
Observers of young children in ECT-I classrooms should f, blend into the 
woodwork" to avoid distracting those they observe. They must be skilled 
listeners and watchers, accurately recording what they see and hear. (See 
Burges, 1976, for further discussion.) 
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Product Evaluation 

The data in a product evaluation usually consist of students' scores 
on norm- or criterion-referenced tests. Teachers and aides are frequently 
called on to collect this information. Unfortunately test administration 
is sometimes haphazard, thereby limiting the validity of the data and the 
usefulness of the evaluation. Assuring proper test conditions is thus an 
important consideration. 

The development of CRTs requires specially trained personnel. Sophis- 
ticated psychometric methods are needed to specify knowledge domains, for- 
mulate questions, ensure test reliability, and determine test length. In 
most LEAs, development of local CRTs will require outside help. This is 
not needed if commercial CRTs are used or if less sophisticated, objective- 
referenced tests are created. 

Human resource requirements for analyzing test results can vary con- 
siderably. On the one hand, few local resources are required if the LEA 

buys an analysis service from tes ^companies \_. If commercial test analysis 

is inadequateyor if the test is a locally developed instrument, more sophis 
ticated LEA skills will be required. Program staff will need to collaborat 
with an evaluator or other person with psychometric training in order to 
translate test results into suggestions for program modification. 
Process -Product Evaluation 

Process -product evaluation draws on both structured process data and 
product information. (See the discussion above for data collection con- 
siderations.) In addition, because analysis requires complex statistical 
methods to relate process and product data, expertise in statistics and 
computer-based data analysis is, needed. In LEAs with an evaluation office, 
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these skills are often available. Other LEAs may need to seek outside 
help. 

PROCEDURES FOR CONDUCTING QUANTITATIVE EVALUATIONS 

Procedures for conducting quantitative evaluation by any of the three 
methods described break down into four categories: 

• Determining the focus of the evaluation 

• Choosing or creating data collection instruments 

• Collecting the data 

• Analyzing the results. 

For both structured process evaluation and product evaluation, the first 
three activities are the most important; if good data are collected, data 
analysis often will be relatively straighforward. Process -product evalua- 
tion, by contiast, not only depends on the quality of the data but also 
requires their extensive analysis to produce useful information for program 
improvement . 

Structured Process Evaluation 

This type of evaluation usually begins with a question, hunch, or theory 
about how the progTam can be improved. In our example, student outcomes in- 
dicated that the early childhood program was not as effective as had been 
hoped. One evaluator's experience with time-on-task studies led him to 
suggest a study on the use of classroom time. Framing evaluation questions 
in advance is necessary because the data collection instruments — whether 
they are questionnaires, interview forms, or observation protocols --have 
fixed questions ard responses. Thus, there is little opportunity to re- 
direct the inquiry in midstream if new concerns arise. 

2d 
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Checklists and coded behavior records are common data collection pro- 
cedures. Observers might use a checklist to count occurrences of certain 
classroom behavior, such as the number of times a child's attention wanders 
during a lS-minute group activity. For more detailed information, coded 
behavior records are often used to count all acts that occur within some 
fixed time interval. This observation procedure requires the development 
of a set of general behavior categories, and a set of rules for assigning 
specific behavior to one such category. The definition of categories de- 
pends upon some preconceived notions about what is important to look at. 
The Flanders teacher-student interaction instrument typifies this approach. 
As Figure 1 shows, its focus on the verbal interaction of teacher and child 
is reflected in the structure of its behavior categories.* Other observa- 
tion systems might focus on social interactions among peers and adults, or 
on the use of materials, or space, or some combination of Ihese. 

Structured interviews represent another common data collection proce- 
dure. Again, the progTam staff together with the evaluator must decide what 
information they want to gather. The program theory, program plans and 
activity lists will suggest what information may be needed. For each of 
these areas, specific questions are developed. Next, the interview form 
is created, field tested, and revised where necessary. Finally, the inter- 
view instrument is ready for administration (Burry, 1978) . 

If the questions for structured process evaluation are well thought 
out and if data collection is done carefully, data analysis should be 
straightforward. A simple presentation of the results of interviews or 

* See Flanders (1965) for further discussion. 



Teacher 

Talk 


Indirect 
Influence 


1. Accepts feeling: accepts and clarifies feeling 
tone of the students in a non-threatening manner. 
Feelings may be positive or negative. Predicting 
and recalling feelings are included. 

2. Praises or encourages: praises or encourages 
student action or behavior. Jokes that release* 
tension, not at the expense of another individ- 
ual, nodding head or saying "uhhuh?" or "go on" . 
are included. 

3. Accepts or uses ideas of student: clarifying, 
building, or developing ideas or suggestions 
by a student. As teacher brings more of his 
own ideas into play, shift to category 5. 

4. Asks questions: asking a question about content 
or procedure with the intent that a student 
answer. 


Direct 
Influence 


5. Lectures: giving facts or opinions about content 
or procedure; expressing his own ideas; asking 
rhetorical questions. 

6. Gives directions: directions, commands, or orders 
with which a student is expected to comply. 

7. Criticizes or justifies authority: statements 
intended to change student behavior from non- 
acceptable pattern; bawling someone out; stating 
why the teacher is doing what he is doing, 
extreme self-reference. 


Student 
Talk 


8. Student talk-response: talk by students in 
response to teacher. Teacher initiates the 
contact or solicits student statement. 

9. Student talk-initiation: talk by students, 
which they initiate. If "calling on" student 
is only to indicate who may talk next, observer 
must decide whether student wanted to talk. 

If he did, use this category. 




10. Silence or confusion: pauses, short periods of 
silence, and periods of confusion in which com- 
munication cannot be understood by the observer. 



(Source: Tuckman, 1972) 



Figure 1: Summary of Categories for Interaction Analysis 
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observations can be instrumental in identifying areas for program improve- 
ment. For example, one study of the use of class time used pie charts to 
display the amount of instructional versus non-instructional time. The 
surprise and consternation about how little time was spent on teaching and 
learning led to efforts that increased the time devoted to teaching and 
reduced other activities.* 
Product Evaluation 

Care in selecting or developing an appropriate test is essential to 
the success of product evaluation. The usefulness of the data depends on 
a precise explication of program concerns and subsequent research questions, 
choosing or creating a test to match these needs, and careful data collection 
Commercial tests may not be pargeted at areas of local interest; early child- 
hood tests, for example, of^en provide little help for assessing the socio- 
emotional goals of a program. Thus, ECT-I staff should carefully examine 
the content of any test to ensure that it measures what they want to evaluate 
If commercial tests are unsatisfactory, the staff may consider creating a 
local instrument; but this can be expensive and time-consuming. 

Similarly, exercising care in test administration is particularly 
important with young children. The resource book Assessment in Early Child* 
hood Education by Haney and Gelberg (1980) discusses this question in some 
detail. In addition, the Standards for Evaluation and Psychological Tests 
(APA, AERA, § NCME, 1974) is a good guide to appropriate practices in test 
administration. 

If these suggestions about test selection and administration are heeded, 
analysis of the test data should be quite straightforward and informative. 



* For a discussion of this evaluation, see Holley (1978). 
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As we saw in our example of product evaluation, simple tables of test re- 
sults can dramatically reveal both program strengths and areas needing 
improvement . 

Process-Product Evaluation 

Process-product evaluation, as we said earlier, combines structured 
process data with the examination of outcome measures to find ways to im- 
prove the program. Our comments on conducting structured process evalua- 
tions and product analyses apply equally to process-product studies. Evalu- 
ation questions must be framed in advance; data collection instruments must 
be chosen or created that are congruent with the evaluation issues; and 
process and product data need to be carefully collected. 

Data analysis, however, plays a larger part in process-product evalua- 
tion than in either of the other two quantitative methods* Identifying the 
relationship between process data from questionnaires, interviews, and ob- 
servations on the one hand and from test scores on the other requires sophis- 
ticated statistical techniques. Further, ECT-I personnel need to interpret 
and use* the results with caution. Statistical analysis may indicate that 
some set of program variables is associated with higher test scores, but 
this does not necessarily mean that the higher scores are due to these 
factors. For example, a process -product evaluation might show that children 
who attend full-day kindergarten do worse in the first grade than those 
attending only half days. But before concluding that the full-day program 
should be modified, other hypotheses should be examined. For instance, are 
children who are expected to do poorly in school selected for the full-day 
sessions? If so, it may not make sense to compare the test scores of chil- 
dren across these two programs. To draw valid inferences here requires 
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both a knowledge of the local program and its context, and statistical 
expertise. Smaller LEAs can consult a statistician or outside evaluator 
to assist in such an effort. 
FURTHER CONSIDERATIONS 

In deciding to use one of these methods, LEA personnel should consider 
several issues. In general, program staff need a clear idea of the questions 
they want to investigate, since the structured data collection procedures 
required here must be created or selected before the research begins. In 
our example" of a structured process evaluation, for instance, the evaluators 
wanted to examine classroom disruptions; therefore, they found instruments 
that coincided with this intention. ECT-I staff should also realize that 
not ^11 aspects of the program can be examined with quantitative methods. 
For 7 instance, since interviews and observations are usually brief and take 
place during one or a few sessions, these techniques are not well suited for 
investigating program interactions in depth. The qualitative techniques 
discussed in the next chapter are more responsive in this respect. 

NOTES ON SOURCES OF FURTHER INFORMATION 

Structured Process Evaluation 

Many authors discuss evaluation methods that are similar to structured 

process evaluation, although they often call their approaches by different 

names and have somewhat different purposes in mind. Some treat formative 

evaluation (for example, Baker, 1974; Scriven, 1966; and Weiss, 1972). 

Others talk about examining program implementation (see Hall & Loucks, 1977; 

Moiris § Fitz-Gibbon, 1978; Rossi, Freeman, 5 Wright, 1979; and Wolf, 19T3) . 

All of these sources contain potentially helpful advice for carrying out a 

structured process evaluation. 
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In addition, many sources discuss the data collection techniques used 
in a structured process study. Maccoby and Maccoby (19S4) , Cannell and Kahn 
(19S3), and Sellitz, Wrightman, and Cook (1976, Chapter 9) contain good dis- 
cussions of interviewing techniques. Medley and Mitzel (1963), Weick 11968) , 
Rosenshine and Furst (1973), Cassel C1978} , and Stalling? (1W7) outline 
observational strategies that are applicable to structured process evaluation. 
Product Evaluation 

Innumerable books and articles discuss *ests and measures. Here we list 
some that are most relevant to interpreting product evaluations. Several 
resource books in this series on ECT-I evaluation present useful discussions. 
Haney and Gelberg (1980) examine issues in the assessment of young children, 
Haney (1980) outlines short-term impact strategies. Kennedy (1980) looks at 
longitudinal data as a means for evaluating ECT-I programs. Other works on 
tests and measures in an evaluation context are Sax (1974), Millman Q974) , 
Airasian and Madaus (1972), and Morris and Fitz-Gibbon (1978). 
Process -Product Evaluation 

In addition to the sources mentioned, Walberg (1974), Cooley (1978), 
and Pedhazur (1975) discuss practical and analytical issues related to 
process -product research. The National Day Care Study (Roupp, Travers, 
Glantz, $ Coelen, 1979) is a good example of this approach. 
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IV. QUALITATIVE METHODS 

This section describes three qualitative inquiry methods: 

• Documentation 

• Investigative evaluation 

• Ethnography • 

These methods resemble each other (and differ from others) in that they are 
based on the idea of f, the evaluator as instrument." Less emphasis is placed 
on standardizing data collection activities, and much is left to the indivi- 
dual evaluator to decide as the work proceeds. Because so much is left to 
the evaluator 1 3 discretion* the choice of person or persons to conduct the 
study is critical. 

Qualitative methods are very flexible, powerful research techniques 
capable of in-depth, broad-based description and analysis. Interpretation 
of early data can substantially influence subsequent research activities. 
Although the three methods are similar in some ways, they differ in the 
time they require, documentation taking the least time and ethnography the 
most. They also emphasize different data collection methods. Documentation 
depends mostly on record examination; investigative evaluation on intensive 
interviews; and ethnography on observation. 

DOCUMENTAT ION 
^ 

Documentation aims at the accurate depiction of ECT-I programs. The 
documentors attempt to collect and examine evidence in order to portray 
the program clearly and truthfully. Documents may include funding appli- ' 
cations, teachers' plans and self -reports , interviews, questionnaires, 
notes from observations, examples of children's work, photographs, and 
diagrams of classrooms. 

3c> 
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Documentation can portray the program in depth by describing repre- 
sentative pieces of it. It can also assess whether the program has been 
implemented as planned by juxtaposing planning documents and testimony 
about what the program should be, and data on what the program actually is. 
INVESTIGATIVE EVALUATION 

This method uses intensive interviewing, observation, and document 
examination. We call this investigative evaluation because it resembles 
detective work or investigative reporting. The evaluator forms hunches, 
follows leads, verifies sources, and throws out hunches that can't be sup- 
ported* This sort of evaluation can address a variety of questions. In 
examining program operations, it might assess what the program does, how 
it works, what its goals are, and whether these goals have changed. In 
looking at implementation, the evaluator might focus on the congruity 
between goals and program reality, and between legal mandates and program 
practice. In suggesting program improvements, investigative evaluation 
can help determine what went wrong and why, and offer recommendations for 
change. 

ETHNOGRAPHIC EVALUATION 

Ethnographic evaluation requires long-term in-depth observation of 
ECT-I programs. Through such extensive observations, the ethnographer 
attempts to understand the program and the meaning of the events to the 
various participants such as parents, children, and staff. In the course 
of these observations, the ethnographer might be interested in issues such 
as how the program operates in different school situations, what those 
involved with the program identify as its strengths and weaknesses, and 
how the program influences young children's school experiences. Such an 
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evaluation can provide a rich description and portrayal of what it is like- 
to participate in the program. Finally, ethnographic evaluation attempts 
to discern the chief features and processes of programs and to provide a 
comprehensive view of their operation, 
EXAMPLES OF QUALITATIVE METHODS 
Documenting a Marine Science Program 

Documentation was used to examine a program in marine science for 
children in kindergarten through fifth grade. This cooperative project 
between an aquarium and a magnet elementary school was intended to increase 
vocabulary, improve observation and reasoning skills, stimulate art work, 
and build interest in science. Aquarium staff, the program director, and 
teachers wanted to know how well the program was doing and how activities 
and processes could be improved. 

By talking with staff, teachers, parents, and administrators the docu- 
mentor determined that program administrators were the main audience for the 
evaluation, but that the aquarium staff, teachers, parents, and state and 
local officials were also interested parties. In their first meeting, the 
documentor and the project director framed evaluation questions based on 
program objectives. Program staff wanted evidence of such things as in- 
creased vocabulary, improved observation, and interest in science. They 
were also concerned about improving discipline at the aquarium. Data gath- 
ering procedures were then matched with each question. For example, in- 
terest in science was assessed by interviewing children and their parents. 
Observing at the aquarium and interviewing teachers provided data about 
children's behavior. Also at the first meeting, agreements were Aade about 
who would collect what data. The documentor offered to make observations 
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and conduct interviews; teachers volunteered to keep diaries; parents and 
aides agreed to take photographs and to collect examples of children's work 

During the evaluation, the documentor tried to get a wide and represen 
tative view of the program* She talked with several groups, including 
teachers, administrators, parents, students, and aquarium staff; sought out 
people with different viewpoints on the project; and made sure that all 
perspectives were included in the final report. 

The documentor communicated her findings in two ways. First, after 
observing activities and interviewing staff members , she discussed her 
reactions with them. Her comments were as descriptive and non judgmental 
as possible. For example, she told several teachers that, when they left 
their rooms during class to have coffee, instruction and discipline broke 
down. On the basis of this information, school staff worked out a plan for 
some teachers to supervise classes while others took their breaks. 

Second, at the end of the evaluation she produced a written report 
that was discributed to all interested parties. This final report was 
descriptive, but here judgments were more explicit. One finding indicated 
that although the program encouraged observation and reasoning both at the 
aquarium and at school, some children were not as involved as one would 
hope. The documentor substantiated this with descriptions from her ob- 
servations, excerpts from interviews, and photographs. She recommended 
that teachers, not aquarium staff, take charge of maintaining order during 
field trips. 
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Investigative Evaluation of a Bilingual Progrsia* 

This evaluation focused on implementation of a California bilingual 
educatijn law that mandates programs for limited-English-speaking CLES) 
and con-English-speaking (NES) pupils. Since new bilingual legislation 
wa5 being considered, the evaluators first determined what issues concerned 
their chief client, the legislature. In consultation with individual 
legislators, a set of key questions was identified: How are children 
selected for bilingual programs, are state mandates for bilingual classes 
—such as prohibiting segregation of NES children— being followed, 0 and are 
the required annual evaluations jseful? 

< 

The evaluators visited 17 school districts, and altogether 19 schools. 
Districts were so chosen as to ensure variability in grade level, location, 
and bilingual concentration. The evaluators, using primarily unstructured 
interviews, spent a few days talking with key -people in bilingual programs 
in each LEA. These included parents, regular and bilingual teachers, the 
bilingual-program director, and the superintendent. The evaluators col- 
lected the following information: how bilingual pupils are identified 
and assessed, how program staffs- determine when pupils no longer need bi- 
lingual education, how many years pupils participate in programs, what 
other compensatory programs NES and LES children participate in, how LEAs 
evaluate their programs, and how much pupil identification, testing, and 
evaluation cost. 

* Because investigative evaluation is relatively new, there are few ex- 
amples of local use. This example draws from a statewide evaluation 
meant for legislators and their aides (Bissel, Chris tophel, Sequeira, 
and Farias, 1979). It does illustrate, however, the kinds of informa- 
tion such an evaluation can produce, and most of the procedures could 
be used by LEAs. 

10 
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The investigators found several problems with bilingual programs in 
the districts they visited. First, state guidelines for bilingual programs 
were unclear to many local staff members. Second, districts were incurring 
unnecessary costs because of inefficient or unnecessary testing and assess- 
ment. Third, the responsibility for bilingual education was too diffuse 
within the state department of education. Fourth, there were inherent dif- 
ficulties—such as limitations in the state of the art of bilingual testing— 
in meeting some of the statutory requirements. To alleviate these problems, 
the report recommended that the department of education consolidate the 
administration of bilingual education in the state, and identify for the 
legislature those requirements of the law that were difficult or impossible 
to apply. 

An Ethnography of a K-to-5 Computer-Assisted Instruction (CAI) Program 
When the staff of a regional educational agency began a computer, 
assisted learning program for kindergarten and elementary school children 
in several rural mountain towns, they asked two ethnographers to evaluate 
the project. Although they set out to sssess the outcomes of the program, 
the evaluators soon realized that they had tp refocus the evaluation. 
Their central goal became investigating the difficulties that arose when 
sophisticated educational technology was brought into a rural, economically 
deprived region of the country. 

After initial visits to program sites and discussions with project 
participants, the ethnographers formulated four areas to investigate: the 
impact of CAI on teachers 1 decision making, on teacher -student relations, 
on classroom social structure, and on children's behavior and personality. 
As the evaluators later admitted, these four areas only roughly guided the 
actual evaluation. 

o 41 
ERJC 



-37- 

e 

During the first year of the project, the evaluators spent 94 person- 
days in the field in intermittent two- and three-day visits. They kept 
three kinds of records: on-the-spot observations of such events as children 
working at the computer terminals; summaries of observations, informal in- 
terviews, and discussions; and documents such as computer printouts. Alto- 
gether, they collected 900 pages of raw data, which they eventually used 
to describe and evaluate the program, 

A brief summary of the results losses the flavor and fullness of their 
evaluation; we therefore present a few excerpts instead. The evaluators 
grouped their findings under three broad categories;' technical difficulties, 
teacher use of CAI, and pupil behavior at the terminals. Regarding the 
first, they concluded that the system was never fully operational, leading 
to continuing teacher and pupil frustration throughout the year. They re- 
lated vignettes from their field notes to substantiate this conclusion. 
Perhaps the most poignant episode concerned the girl whose printout read 
"Cry again" instead of "Try again, " Her response to her teacher was, M 0h, 
Mrs. Martin, I could just cry and cry again" (p, 7). 

Teacher use of the system varied considerably. For example, the evalu- 
ators noticed wide differences in the integration of CAI with classroom 
instruction. One excerpt from their field notes illustrates the lack of 
integration: 

I made a specific point of checking with two of the [first- 
grade] girls about where they were currently in their math 
lessons. I wondered how closely the drills that they were 
taking corresponded to what they were doing in class. Ap- 
parently it is not very close. Ruth told me that the day's 
classroom lesson was on "writing mathematical sentences." 
The drills, however, were all simple addition and sub- 
traction problems, (p. 20) 
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The findings on pupils 1 behavior at the terminals are the most intri- 
guing aspects of the evaluation. In general, the evaluators discovered 
complex interactions between children and computer and among children work- 
ing at the terminals. For example, they found that CAI generated unanti- 
cipated competition in some classrooms. 

Midway through the morning I happened to notice three boys 
working on the terminals. They made an effort to start 
together, and it was a real contest. It should be noted 
that the three boys were not on the same lesson. Never- 
theless, there was a great deal of competition to see CI) 
who would finish first, and (2) who would get the highest 
percentage. The boy who finally did finish first raised 
his arms above his head like a boxer and crowed rather 
exaltedly, "1 won, I won." The sweet smell of success was 
even greater when he found out that he had achieved a 
higher percentage score on his test than either of his two 
buddies. Both of them looked a little bit crestfallen, 
particularly the boy who ended up last. (p. 31) 

These excerpts- illustrate the kind of description characteristic, of ethno- 
graphy, which can lead the reader to a deeper understanding of what the 
program in operation was like. We encourage- anyone with further interest 
in ethnographic techniques to read the whole evaluation report CSmith and 
Pohland, 1974). 

PROCEDURES FOR CONDUCTING QUALITATIVE EVALUATIONS 
Human Resources 

Documentation . The documentation of the marine science program illus- 
trates how several groups can be drawn into the evaluation process. Teachers 
might take photographs of the classroom and special .activities , keep folders 
of children's work, and record anecdotes. Parents might keep time-activity 
charts, write up. classroom notes, summarize questionnaires, and take pictures 
Aides might keep folders on children's work up to date, make diagrams of the 
classroom, and collect statistical information. Administrators can provide 
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an overview of the program and information about its environment and context. 
The outside documentor organizes the documentation, creates information 
gathering systems, observes the program, analyzes data, and makes recommen- 
dations for improvement. 

Involving teachers, aides, parents, and administrators as well as a 
trained evaluator is a chief strength of documentation. Because several 
groups generate and collect documents, diversified views on the program 
are provided. The use of program staff and parents to collect data can 
also help to reduce evaluation costs. 

The documentor is, of course, the key person in documentation. Since 
documentation is not one procedure but several, documentors must tailor the 
techniques to the particular situation. Different data must be collected to 
assess different goals and to answer different questions. As the marine 
science documentation illustrated, the documentor must be familiar with a 
wide variety of evaluation tools and know which are best for answering a 
particular question. 

To document a program fairly and accurately requires a good deal of 
sensitivity and judgment. The documentor of the marine science program, 
for example, took pains to make sure that all opinions about the program 
were represented in the final report. Only a skilled documentor can keep 
the process from threatening those it is meant to enlighten. Certainly, 
keeping the evaluation factual rather than judgmental, at least in the 
initial stages, helps to promote this goal. At the same time, facts them- 
selves can be threatening, especially when they inform people that what is 
happening is not what they want to happen. If care is not exercised, the 
focus of the evaluation can easily shift from learning about a progTam tc 
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fault-finding and excuse-making . Further, the links between the documents 
collected and program problems are not always clear; nor are strategies 
for program improvement always self-evident from the documentation report. 
Thus the documentor's knowledge about the program and experience in educa- 
tional contexts are important assets for teasing out relationships and sug- 
gesting improvements. 

Investigative evaluation . Unlike structured interviews and observations, 
the intensive interviewing involved in investigative evaluation is much like 
the work of a journalist or a criminal investigator. Good intensive inter- 
viewing depends not only on understanding interview techniques but on adapt- 
ability to contingencies, good guesswork, and intuition. The quality of an 
investigative evaluation thus depends on the skepticism, instinct, integrity, 
intelligence, and to some extent luck of the evaluator. 

Ethnographic evaluation . As one can infer from the ethnographic study 
of the CAI program, ethnographic evaluations require experienced observers. 
Observers can be either nonparticipants, passively observing children and 

■v 

staff, or participants, actually becoming part of what they observe. Both 
evaluators of the computer-assisted instruction program were nonparticipant 
observers. In another evaluation, Smith and Geoffrey (1968) used both 
participant and nonparticipant techniques to study an urban classroom. 
Geoffrey, the classroom teacher and participant observer, recorded at the 
end of each day his perceptions or what went on in class. Smith, the non- 
participant, observed the class and kept detailed notes on what he saw. 
Thus the evaluators obtained both inside and outside views of what took 
place and why. 
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Schatzman and Strauss (1973) provide a good impressionistic description 
of an ethnographer: 

[He] is a learner, has patience, is tolerant and sympathetic. 
He wonders first and. judges last; he appears to be that way 
and is_ that way. Furthermore, he generally accepts whatever 
he sees and hears at face value; he denigrates no motives. 
He does not visibly take sides on arguments ... no matter 
how much he may be invited to do so. He is open to the dis- 
covery of whatever is not so obvious to others. He is most 
considerate, polite, but not shy; he is, in fact, rather 
tough in the sense that he cannot be put off for too long, 
nor shamed or coerced.: He cannot be bought off or drawn 
into private arrangements, even to gain the data he needs. 
He assumes that the hosts ultimately would have it no 
other way. (p. 65) 

Qualitative Data Collection and Analysis 

Documentation, investigative evaluation, and ethnographic evaluation 
depend primarily on document 1 examination, unstructured interviews, and direct 
observations. Although all three methods may employ all these procedures 
to some degree, record examination is the chief procedure for documentation, 
intensive interviewing for investigative evaluation, and long-term observation 
for ethnographic study. 

Documentation: Examining records . The documentor can obtain informa- 
tion by systematically collecting existing records and by creating procedures 
to gather additional information not currently available. It is often use- 
ful to begin by simply listing the most important activities of the program, 
and the records that are normally maintained by program staff. When existing 
records provide insufficient information about particular activities, new 
procedures must be devised to gather such data. Figure 2 shows a chart of 
activities and records from a first-grade documentation; for each activity 
of the program, the documentor identified existing records or suggested new 
record-keeping practices where necessary. 
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There azte several ways to analyze and present results from classroom 
documentation. Observers can summarize and transform their notes and check- 
lists into a narrative of what occurred in the classroom while they were 
present. Such a narrative helps describe how the program appeared to the 
observer: 

9:00 Skills Time: Children arrive, sign up for lunch, check 
the mini-lab board to choose an activity for the end of the 
day, and gather in two groups for a short meeting to plan 
the day. 

The S and 6 year olds choose from a variety of activities, 
and meet in small groups or individually with the teachers 
to work on language experience activities, reading, or number 
concepts . 

The 7-9 year olds work in their individual study carrels with 
partners at reading, writing, math or current projects. They 
confer individually with teachers to plan their individual 
goals or projects. Parents help individuals or small groups. 
Another teacher or aide gathers small groups for skills se- 
quence teaching or listens to children read from their self- 
chosen books. 

At 10:30 the groups begin to flow from room to room as the 
children work on self -chosen activities at the centers in 
4 the school. They may cook, do a, play, work at the science 
center, listen to recorded stories, paint, play games, or 
continue on M school work." Skills centers are assigned 
weekly to the 7-9 year olds in the areas of writing, math, 
and reading. The groups gather again for a story and dis- 
cussion period at the end of the morning, and to hear their 
current story be read aloud. 

(Engel, 197S, p. 61) 
Categorizing observations can also be useful for analysis and presen- 
tation. For example, a visitor can observe a teacher for one hour, noting 
all his activities and categorizing them according to purpose. Data of 
this kind illuminate a variety of questions such as how teachers spend their 
time. In addition, graphic displays can assist in analysis. For example, 
the evaluator can observe children's choice of activity for a set time, 
determine the frequencies of choice, and graph them. Graphs can help 
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Activities 
(First Grade) 



Record or Document to be 
Used for Monitoring the 
Activity, and Assessment 
of its Adequacy 



Reading Aloud with 
Teacher or Aide 
(3 times per week) 

OR 

Reading Aloud into a 
Tape Recorder at 
Recorder Corner 
(3 times per week) 



Reading Seatwork: 
Choice of 
- Workbook 



- Library book reading 



Perceptual -Motor Time, 
e.g. IS minutes per day 
in school gym with aide 



Clapping rhythm 
exercise (in group) 



Open balance 
(individual, on 
balance beam, 
jungle gym, etc.) 



Teacher/aide's record 
book: gives dates of 
reading and number of 
pages read ' 
ADEQUATE 

Aide's recording form: 
gives amount of time, 
progress, distractions 
ADEQUATE 



Completed workbooks 
record of pace and 
completion 
ADEQUATE 

Library book use 
INADEQUATE 



No records 
INADEQUATE 

(Have aide keep a check- 
list of length and con- 
tent of daily sessions) 

None 

INADEOUATE 

(Ask aide to keep notes 
on this activity) 

None 

INADEQUATE 

(Ask aide to keep notes 
on this activity) 



Regularity of 
Document Collection 
and Assessment of 
its Adequacy 

Constant 

documentation 

ADEQUATE 



Only on discipline 

problems 

INADEOUATE 

(Ask staff to keep 
records on all students) 



Constant documentation 
ADEQUATE 



No documentation 
INADEQUATE 

(Have students record 
library books read) 



(Adapted frok Burry, 1978, pp. 12-13) 



Figure 2: ProgTam Activities and Records 
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depict the popularity of classroom activities for each child and for the 
whole class. 

Investigative evaluation: Intensive interviewing . The central proce- 
dure for investigative evaluation— intensive interviewing — is most useful 
when neither the interviewer's exact questions nor the respondent's likely 
answers can be specified in advance. Collecting data by this procedure 
seems particularly well suited for discovering the perceptions, attitudes, 
and motivations that, when sought explicitly, respondents may be unable or 
unwilling to provide. 

The investigator begins the evaluation by "scbuting the scene"— reading 
all available program documents, and doing preliminary interviews, often by 
phone. This background work helps the investigator to become familiar with 
the program, to identify key individuals, informants, and sites, to begin 
formulating key questions, and to develop hunches and initial perspectives 
on program problems. 

The investigator is then ready to begin intensive interviews with key 
people. Intensive interviewing is quite different from the structured, 
standardized interviews we have already discussed. Rather than asking every- 
one the same questions, the interviewer tailors questions to each person and 
situation. What questions are asked and how they are phrased depends on the 
evaluation issues, the circumstances of the interview, and the respondent's 
reaction to previous questions. Intensive interviews are aimed both at col- 
lecting valid, factual information, and at gathering remarks and anecdotes 
that reflect views of the nrogram in action. 

Data collection and analysis in investigative evaluation are closely 
linked. \ These two activites go on aLcost simultaneously: preliminary 
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analysis indicates additional data to collect; as further data are collected 
the evaluation's hypotheses can be further examined. In short, data analysis 
goes on throughout the evaluation, even during report writing. While collecting 
data, the investigator forms and checks out hunches about program facts, fre- 
quencies, and causations. He corroborates facts by looking for confirming 
and contradictory evidence. If he finds substantial evidence consistent with 
the hunch, his confidence increases; contrary evidence leads h^m to discard 
the hunch. 

Ethnograp hy: Long-term observation . In some ways, long-term observation 
resembles intensive' interviewing. The evaluator maintains considerable flex- 
ibility in determining what is to be observed, and the technique is particu- 
larly useful when complex behavior is to be studied. As with intensive inter- 
viewing, background work is a prerequisite of data collection- Similarly, 
the pieces of information collected through observation are not automatically 
accepted as representative or valid; rather, the ethnographer attempts to 
verify what he sees by observing many situations, just as the intensive in- 
terviewer checks out information obtained from each respondent^. 

The initial step in ethnographic data collection is deciding what to ob- 
serve. «Just as the investigative evaluator scouts the scene, the ethnographer 
often begins with some -unfocused observations. He views as many aspects of 
the program as possible and "over-records" his observations. As time passes, 
he begins to identify representative times, people, and situations, and to 
form hunches and hypotheses about what happens and why. Ideally, there is 
a pattern to this process: the initial focus on very general observations 
spirals inward to increasingly specific observations of key events, relation- 
ships, and the like. The ethnographic evaluation of the CAI progTam illustrates 




this. The evaluators initially focused on broad topics such as teacher decision 
making and classroom social structure. By the time. they had finished the 
evaluation, they reported on more specific topics such as technial difficulties 
in implementing the program and children's behavior at the computer terminals. 

During both general and focused observations, field notes are the main 
data collection device. These notes are detailed descriptions of settings, 
activities, and people. The observer records or paraphrases conversations. 
He may note individual characteristics, idiosyncracies, and non-verbal be- 
havior. In addition, he writes up his tentative interpretation of what he 
sees, including its significance; whether it suggests hypotheses, and whether 
it confirms or contradicts other data or earlier hunches. As we saw in the 
example above, field notes are extensively quoted in the evaluation report 
to substantiate conclusions and recommendations. 

Data analysis in ethnographic evaluations resembles the analytic pro- 
cedures of investigative evaluation. As data collection takes place, the 
observer or other staff members r^ad through field notes trying to spot trends, 
categorize the data, and generate hypotheses. The results of this analysis 
influence subsequent data collection, which in turn forms the basis for fur- 
ther analysis. As this process continues, the research becomes more focused, 
and ultimately the report framework emerges. The details of this process can 
vary substantially from investigator to investigator, which brings a very 
personal character to such qualitative research. 
FURTHER CONSIDERATIONS 

Perhaps the single most important decision in qualitative inquiry is the 
selection of the evaluator. How perceptive and credible the study is hinges 
on the skill, integrity, and perspective that the evaluator brings to it. 
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In a real sense, "the evaluator as instrument" represents both the great 
strength of these techniques and potentially their greatest problem. In 
reading a qualitative evaluation report, we should continually ask ourselves, 
"Whose view of the program are we really seeing?" 

The controlling of personal bias is a vital concern in qualitative in- 
quiry. Both the evaluator and the subjects of the study, teachers and chil- 
dren, are potential sources of bias. The evaluator 's preconceived ideas of 
how things work and what are good and poor eduational practices can so distort 
what he sees and hears that what he reports may be little related to what 
happened. Similarly, subjects can produce biases and errors, either because 
they do not want to cooperate or because they do not know important information. 
They can bias observations by "masking" their behavior from the evaluator— 
that is, behaving differently because an observer is present. The professional 
qualitative evaluator is aware of this and attempts to safeguard against it 
in conducting the research. Persons of considerable skill and experience are 
required here, and qualitative evaluation should not be attempted without them. 
NOTES ON SOURCES OF FURTHER INFORMATION 

There are several general sources on qualitative research, including: 
Denzin (1970), Filstead (1970), Guba (1978), Lofland (1971), and Patton (1980). 

Documentation . Carini (1975) and Perrone (1977) discuss the rationale 
for documentation. Perrone, in particular, looks at strengths and weaknesses 
of this approach. Engel (1975, 1977), Heia "(1975) , Burry (1978), and Suarez 
and Vandivere (1978) discuss how to go about documenting prbgrams. Each 
author's work is aimed at different programs and audiences. For example, 
Hein looks at open education, and Suarez and Vandivere treat programs for 
preschool handicapped children. 
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Investigative evaluation . Because this is a new approach to evaluation, 
there are fewer sources and examples than for other methods, ^fu^phy C1980) 
is a good guide for carrying out this kind of evaluation, Douglas Q976) 
discusses investigation methods, but because his research interests are far 
removed from classrooms, his book may be less useful than Murphy's, The 
Report Manual from the California Office of the Auditor General C1978) pre- 
sents another perspective on investigative methods, Bissell Q979) also dis- 
cusses this procedure, and reports by her and her staff are good examples of 
one form of investigative evaluation (e.g., Bissell, Potter, Barber, § Sheperd, 
1978; Bissell, Potter, Herdell, § Tamayo, 1979), 

Ethnographic evaluation . There is a large body of writings on educational 
anthropology and ethnography. Less has been written on applying these ap- 
proaches to evaluation. General works of some applicability are Bruyn C1966) , 
Good and Brophy (1978), Schatzman and Strauss (1973), McCall and Simmons C1969), 
and Rist (1975). Authors who discuss evaluation applications are Smith C1979) , 
Stake (1978a), Berliner and Tikunoff (1977), and Jackson (1974), Wolcott 
(197S) and Muihauser (1975) point to some problems in applying these methods 
to evaluation. Several good examples of this research exist that are evalua- 
tions or are much like evaluations; see Jackson (1968), Stake (1978b), Tikunoff, 
Berliner, and Rist (1975), Smith and Geoffrey (1968), and Rist (1973). 
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V. DEVELOPING AN EVALUATION PLAN • 

COMPARING THE METHODS 

We have introduced in this resource book several evaluation techniques 
that hold considerable promise for generating information that can be useful 
to LEA staff in attempting to improve their programs. Table 4 provides a 
quick overview of the methods discussed and some of their distinctive charac- 
teristics. They vary considerably in the kinds of question they address, in 
the likely audience for the results, in the data collection methods emphasized, 
and in the human and physical resource requirements. 

Although all the methods can address a wide range of questions, each is 
better suited to some .questions than' to others. ' Self-study and outside review 
typically focus on how well the program measures up to some set of standards 
or expectations of program performance. Structured process evaluation seems 
best suited to answering straightforward implementation questions, such as 
whether new curricular material is used in classrooms. Qualitative methods 
can investigate quite complex questions about human behavior and interaction 
in school settings. 

As a result of. the difference in emphasis, the audience for each method 
>will also- vary. The self-study report is aimed primarily at the program^ 
staff. Information from structured process evaluation may be most useful to 
those program administrators who have little direct contact with their program 
and desire some basic information about "what's going on." Outside review 
and investigative evaluations produce information that school boards, PACs, 
and program policy groups are likely to find most interesting. 

As for data collection, although many of the methods use some form of 
observation, interviews, and document examination, each tends to rely on some 
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Table 4 Comparison of ECTO 



Improvement 
Method 



Brief Description of Method 



Human Resource Requirements 



Self-Study Committees of the staff compare 
aspects of program to some set 
of standards or expectation of 
the program in order to identify 
strengths and weaknesses 



Teachers and other staff 
members collect and' analyze 
data and write reports 



Outside Visiting committee examines the 

Review program to help ECT-I staff 

identify program strengths and 

weaknesses 



Staff from nearby programs 
and other early childhood 
specialists collect and 
analyze data 



Structured Evaluator ^assesses relatively 
Process straightforward implementation 

Evaluation concerns such as whether materials 
are in classrooms; quite us.eful 
for administrators of large pro- 
grams who may have infrequent 
contact with. classrooms 



Staff members fill out ques- 
tionnaires and respond to 
interviews; local e valuator 
or prbgram assistant collects 
and analyzes data 



Product Evaluator examines tests and 

Evaluation other outcome data to identify 
program strengths and weaknesses 



Staff members collect test 
data; program director or 
evaluator analyzes data 



Process- r Evaluator investigates statistical Teachers and administrators 
Product relationships between various ma y supply some information; 

Evaluation program processes and outcomes observers and interviewers - 

collect process data; statisti- 
cian or evaluator analyzes data 
and interprets results 



Documenta- 


Documentor collects, arranges, 


Teachers, aides, parents, 


tion 


and interprets program documents 


and administrators may col- 




to depict the program Jfor the . 
sts'tf, administrators, and 


lect program documents; 




trained documentor directs 




parents 


the evaluation 


Investiga- 


Evaluator uses interviews and 


Highly trained outside 


tive , 


qualitative techniques to investi- 


evaluator conducts the 


Evaluation 


gate implementation and compliance 
questions for bodies such as 
school boards, state legislature, 
and Parent Advisory Comrtittees 


study 
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Ethnographic 
Evaluation 



Ethnographer examines fine- 
grained interactions using 55 
anthropological and ethno- 
methodological perspectives 



Highly trained qualitative 
investigator conducts the 
evaluation 
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Primary Data T i me Con . 

Collection Techniques siderations Additional Considerations 



Document * examination; 
brief interviews; 
questionnaires 



Time-consuming — 
may require several 
months 



Can be, and often is, used to 
inform an outside review 



Document examination; 
checklists; rating 
scales; informal 
interviews 



'Relatively quick by 
itself, but it can 
be time consuming 
'if conducted in con- 
junction with other 
efforts such as 
self-study 



A chief benefit may be dissemina- 
tion of ideas that outside 
reviewers bring to and take home 
from the program 



Structured observa- 
tion; structured 
interviews 



Relatively quick 



Can be a very useful program 
management tool and is easy to 
implement 



Norm- and criterion- 
referenced tests 



Relatively quick 
1 assuming that a 
short-term impact 
evaluation is 
already being done 



Helps in identifying where to/ 
make improvements but provides 
little information on how 



Structured observa- 
tion; structured 
interviews; question- 
naires; standardized 
tests 



Relatively time- 
consuming 



Explicitly ties program attributes 
to outcomes; -requires standardized 
measurement and therefore may 
inadvertently focus attention on 
easily measured but less important 
program variables 



Document examination ;** Relatively quick 

self -reports; 

interviews ; 

questionnaires; 

observation 



Depends on available records, 
whose quality may \'ary greatly 



Intensive interviews; Relatively quick 
do^urent Examination; 
transient observation 



A new approach, as yet rarely 
applied to program evaluation; 
may raise ethical problems 



O . Interviews; long- 
ERJC term observation; 
Ha ™ B ™ document examination 



Relatively time- 
consuming 
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The description and analysis this 
method allows is a real strength; 
availability of a qualified 
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data collection procedures more than on others. For example, investigative 
evaluation uses primarily intensive interviewing, while ethnographic evalua- 
tion depends chiefly on observation; but the former also uses observations, 
.and the latter, interviews. The choice of data collection procedures in part 
determines human resource requirements. Thus self-study, outside review, 
structured process evaluation, and product evaluation, which all use relatively 
structured procedures, can be done by persons who have little formal evalua- 
tion training. On the other hand, investigative evaluation, documentation, 
and ethnographic evaluation, which depend on open-ended techniques, require 
highly trained researchers. 

Finally, the time needed for evaluation also differentiates the methods. 
Some approaches can provide information in a few weeks or months; others can 
take a year or more. Outside reviewers, for example, can examine, a progTam 
in one week or less and return their suggested modifications a month later. 
On the other hand, an ethnographic study of a large program could take a year 
for data collection and another year for analysis. 

GETTING STARTED: SELECTING AN APPROACH 

As we have tried to illustrate throughout the resource book, no single 

evaluation method is always most appropriate. Rather, trade-offs between 

strengths and weaknesses make each more appropriate in some situations than in 

others. In selecting an approach, a first consideration is simply resources. 

If few resources are available for evaluation, or if most evaluation resources 

are allocated to other activities, methods such as process -product studies and 

ethnographic evaluation which tend to be resource intensive may not be feasible. 

Moving beyond the resources issues, the neat question is, "What is it 

that you want to know about the program?" Are you interested in examining 
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the whole program or just selected features, or even a single topic? An 
answer here usually starts with some impressions about problems with current 
practice, and sone hunches or hypotheses about their causes. Discussions 
with a professional program evaluator will usually result in some suggestions 
for appropriate evaluation methods. For example, a program administrator may 
be troubled because a new reading curriculum doesn't seem to be working. She 
may wonder whether program materials have reached the classrooms, and whether 
teachers are using them regularly. A structured process evaluation probably 
can answer these questions. Evaluators can investigate how many teachers 
have received program materials, and when; and can observe classrooms'to see 
whether the materials are used or are still in boxes. 

In addition to technical considerations of matching an appropriate 
evaluation method to specific research questions, there are very important 
questions of personal interaction. In particular, the perspectives and 
interests of the target audiences should play a key role in developing an 
evaluation plan. If an audience is to find the results of an evaluation 
useful, it must view the research questions and proposed procedures as appro- 
priate, and have confidence in the credibility and integrity of the evaluators 
The involvement of these "stakeholders" throughout the entire evaluation, 
from developing the initial plan to conducting the study and reviewing the 
preliminary findings, seems crucial in engaging their commitment to the 
evaluation and their use of the findings. 
IMPLEMENTING THE PLAN 

Evaluation efforts, like programs themselves, can be quite complex, and 
as a result are sometimes difficult to carry out successfully. Some methods 
entail the collaboration of several people over an extended time. For example 



-S4- 

self-study may require the part-time participation of the whole staff for 
several months. Other methods demand intricate data collection procedures. 
In particular, investigative evaluation, documentation, and ethnographic 
evaluation depend upon careful interviewing, document review, and observation. 
Still others, such as process -product studies, involve complex statistical 
data analysis . 

Staff motivation is, of course, critical- When staff members are fully 
involved as stakeholders in the evaluation they are likely to take the effort 
seriously. If, on the other hand, evaluation is mandated from outside the 
program, ov if a directive comes down that the staff will participate, 
evaluation by any method is likely to be more difficult. Clearly, motivation 
becomes more important in methods that require staff members to collect or 
analyze data. Thus, self -study may fail if the staff believes it is a waste 
of time. More generally, if staff members fear the evaluation, see it as 
intrusive, or think it is unneeded, it seems unlikely that any method will 
be- successful. 

The availability of appropriate evaluation expertise. is also critical. 
As we have pointed out, several methods require ingenuity, intelligence, 
training, and experience on the part of the people performing them. Clearly 
the quality of ethnographic and investigative evaluation, for example, will 
vary enormously depending upon who conducts the study. Similarly, just 
designating someone as the "project documentor" will hardly ensure that a 
good, or even a sufficient, documentation will result. 
CONCLUDING REMARK 

While keeping this resource book brief, we also have tried to make it 
rich in ideas about alternative evaluation methods and to illustrate how 
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their use might ultimately lead to program improvement. What we have 
. presented here, however, is only a sample of what is possible. Further, 
the methods outlined above are not in any sense rigidly defined, but rather 
general approaches that must be tailored to a variety of technical and 
human considerations . An underlying- theme throughout is that there is 
a great potential here which few LEAs have yet tapped. We hope that this 
resource book provides some encouragement in moving in that direction. 
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